fix: build-images CI failures (deadsnakes PPA flakiness + nightly version mismatch) - #1780
Merged
Conversation
… image build - ci/docker/Dockerfile: retry `add-apt-repository ppa:deadsnakes/ppa` up to 5 times with backoff. It calls Launchpad's API to fetch the PPA signing key, which is prone to transient failures (observed HTTP 500 GPGKeyTemporarilyNotFoundError), failing the whole image build. - build_test_publish_images.yaml: compute-cuopt-ver now sets RAPIDS_VERSION_SUFFIX=".post<datetime>" like ci/build_wheel.sh does (since #1661), so the version pinned in the Dockerfile's pip install matches what's actually published for nightly wheels. RAPIDS_VERSION_SUFFIX is a no-op for release builds, so release tagging is unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI Test Summary⏭️ All 5 test job(s) skipped. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughThe Docker build now waits 5 seconds between deadsnakes PPA registration retries. The workflow comment describing nightly version handling is shorter. Workflow behavior remains unchanged. ChangesCI build updates
Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This PR makes localized CI changes to retry a transient package-source failure and align nightly image versioning with published wheels. No actionable merge-blocking risk remains in the supplied evidence; it is merge-ready after normal checks and review. Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tmckayus
approved these changes
Aug 24, 2026
ramakrishnap-nv
added
bug
Collaborator
Author
|
/merge |
3 tasks
ramakrishnap-nv
added a commit
to ramakrishnap-nv/cuopt_public
that referenced
this pull request
Aug 25, 2026
build-images failed with 'Expected environment variable RAPIDS_DATETIME_STRING to be set and non-empty'. rapids-datetime-string does not generate that value in CI -- it only validates that something upstream exported it, and exits 1 otherwise. It is exported by the RAPIDS shared workflows, which is why ci/build_wheel.sh can use it. This image job is a plain ubuntu-latest job that never receives it, and the variable is set nowhere in this repo, so NVIDIA#1780's use of it could not work. Setting it locally would not help either. The suffix must match the timestamp the wheel job used, and a freshly computed one is simply a different unpublished version -- the same 'No matching distribution found' failure NVIDIA#1780 set out to fix. The value is not reachable from here: the wheel jobs are shared-workflow calls and build.yaml exposes no outputs. Resolve the version from the nightly index instead. build-images runs after wheel-publish-*, so the newest .post build of this base version is the one this run produced. All cuopt packages share a single .post number across arches and CUDA suffixes (verified on the index), so one resolved version pins every install in the Dockerfile. This also fails loudly and early when no matching wheel was published, rather than deep inside pip in a matrix job. Release builds are unaffected: they keep the bare version and never consult the index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
ramakrishnap-nv
added a commit
to ramakrishnap-nv/cuopt_public
that referenced
this pull request
Aug 25, 2026
build-images failed with 'Expected environment variable RAPIDS_DATETIME_STRING to be set and non-empty'. rapids-datetime-string does not generate that value in CI, it only validates that something upstream exported it, so NVIDIA#1780's use of it could not work in this job. The value is already available. build.yaml runs compute-build-details.yaml and passes build-datetime to every other build job; build-images was the only one that did not take it. Wire it through and use it directly, which is what NVIDIA#1780 intended. Because the datetime comes from the same build-details job that stamped the published wheels, the pin resolves to this run's build. Resolving the newest matching version from the package index was considered and rejected: versions are keyed on distance from the last tag, so a feature-branch build and a main build can share an alpha, and the lookup could pick the wrong commit's wheels. Release builds are unaffected: RAPIDS_VERSION_SUFFIX is a no-op there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two independent issues causing the
build-imagesworkflow to fail (e.g. https://github.com/NVIDIA/cuopt/actions/runs/32692816239):ci/docker/Dockerfile:add-apt-repository ppa:deadsnakes/ppacalls Launchpad's API to fetch the PPA's signing key, which is subject to transient failures (observedHTTP 500 GPGKeyTemporarilyNotFoundError). Wrapped it in a 5-attempt retry loop with backoff so a transient Launchpad hiccup doesn't fail the whole image build..github/workflows/build_test_publish_images.yaml:compute-cuopt-vercomputedCUOPT_VERwith plainrapids-generate-version, butci/build_wheel.shhas published nightly wheels with a.post<datetime>suffix since ensure nightly builds always produce new packages #1661 (dbaf523f). The two were out of sync, so nightly image builds pinned a version (e.g.26.10.0a43) that was never actually published — only.post-suffixed builds exist on the index — causingpip installto fail with "No matching distribution found". Now setsRAPIDS_VERSION_SUFFIX=".post${RAPIDS_DATETIME_STRING}"before callingrapids-generate-version, mirroringci/build_wheel.sh.RAPIDS_VERSION_SUFFIXis a no-op on release builds (confirmed inrapids-generate-version's source), so release tagging/versioning is unaffected.Test plan
build-imagesworkflow (nightly build) and confirm thepython-envstage installs Python without hitting the deadsnakes PPA errorpip install cuopt-server-cu13==<CUOPT_VER>resolves to a real published nightly wheelIMAGE_TAG_PREFIX) is unchanged for tagged releases🤖 Generated with Claude Code